Papers with multiple-choice question
IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark for LLMs (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models’ referential understanding. |
| Approach: | They propose a benchmark for mention resolution presented in a multiple-choice question format and a curated mixture of different mention types and corresponding entities. |
| Outcome: | The proposed model achieves 81.9% accuracy while the open model achieve 80%. |
Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Many commonsense reasoning questions require a hard selection of a single correct answer . ambiguity and semantic mismatches are common in many MCQs . |
| Approach: | They collect plausibility judgments on 5 000 commonsense reasoning questions . they find that the answer rated most plausible does not match the benchmark gold answers . |
| Outcome: | Experiments with LLMS reveal low accuracy and high variation in performance on the subset . high plausibility rating for the most plausible answer is highlighted in bold . |
Investigating Data Contamination in Modern Benchmarks for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for large language models are inflated and inconsistent with actual performance. |
| Approach: | They propose a retrieval-based system to explore potential overlaps between benchmarks and pretraining corpora and a protocol to investigate testset slot guessing. |
| Outcome: | The proposed method exploits overlaps between evaluation benchmarks and pretraining corpora and masks a wrong answer in a multiple choice question and prompts the model to fill in the gap. |
BnMMLU: Measuring Massive Multitask Language Understanding in Bengali (2026.findings-acl)
Copied to clipboard
| Challenge: | Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize low-resource languages like English. |
| Approach: | They propose a benchmark for massive multitask language understanding in Bengali . they use a dataset that preserves mathematical content via MathML and a subset of questions most frequently missed by top systems to stress difficult cases. |
| Outcome: | The proposed benchmark covers 24 model variants across 11 LLM families. |